Add scaled usage metrics to per-Dandiset table - #241
Open
CodyCBakerPhD wants to merge 12 commits into
Open
Conversation
Normalise the raw per-Dandiset usage numbers by how much content each
Dandiset actually holds, adding four columns to the "Usage per Dandiset"
table: "Bytes / Size" (bytes sent per byte stored), "Views / Asset",
"Downloads / Asset" and "Requests / Asset".
The denominators come from two new data sources fetched from the GitHub
raw CDN, dandi-cache/dandiset-id-to-number-of-assets and
dandi-cache/dandiset-id-to-total-size. Both are gzipped JSONL served
without a Content-Encoding header, so they are inflated client-side via
DecompressionStream. Either fetch failing only leaves the scaled
columns empty; the rest of the page still renders.
A ratio with an unknown or zero denominator ('undetermined', and the
handful of Dandisets absent from the derivatives) renders as "--" and
sorts to the bottom of the table in both directions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts
CodyCBakerPhD
marked this pull request as ready for review
August 10, 2026 20:43
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #241 +/- ##
==========================================
+ Coverage 14.04% 17.82% +3.78%
==========================================
Files 4 4
Lines 1289 1447 +158
Branches 325 371 +46
==========================================
+ Hits 181 258 +77
- Misses 1106 1184 +78
- Partials 2 5 +3
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Fix "normalising" -> "normalizing" in the changelog entry for the scaled metrics, and write the convention down in AGENTS.md so it is not just tribal knowledge. CLAUDE.md is a symlink to AGENTS.md rather than a second copy, so both filenames resolve to one set of instructions (matching how LICENSE already symlinks into LICENSES/). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts
Put the four scaled columns ahead of the raw metrics they are derived from, and rename those to "Total Bytes", "Total Views", "Total Downloads" and "Total Requests" so the two groups read distinctly. Widen the per-Dandiset table's window (1100px -> up to 1600px, still capped by the available width) so all ten columns fit on a wide screen instead of scrolling horizontally. Columns gained an optional `default_sort` flag, set on "Total Bytes": the table sorts by the first numeric column by default, which would otherwise have moved to "Bytes / Size" and left the table ordered differently from the plot beside it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts
Replace the CLAUDE.md -> AGENTS.md symlink with a single real CLAUDE.md holding the instructions, so there is one file rather than two names for it. Nothing outside the changelog referenced AGENTS.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts
Three additions to the per-Dandiset table: - "Total Size" and "Total Assets" columns at the end, so the denominators behind the scaled metrics are visible next to them. Both read "--" for Dandisets missing from the content derivatives. - An "Ignore testing dataset" checkbox that drops Dandisets 000027, 000126 and 000717, whose usage is dominated by automated testing of the archive. It is remembered in the URL, applies to the table (and its download) rather than the plot, and is only shown in the table view of the archive-wide selection, the one place it means anything. - The table now uses the full page width rather than the 1100px cap the other tables keep, so all twelve columns fit on a wide screen. The table is `width: fit-content`, so this is a ceiling, not a stretch. The "Data" menu's "Download raw file" becomes "Download table": the source file no longer contains what the table shows, so the download is now built from the table itself -- same columns, same formatting, same sort order in effect. The source file is still reachable via "View file on GitHub". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts
… sort The setting belongs with the section's other settings, so move "Ignore testing datasets" (plural, tooltip "Hide datasets used for testing purposes") into the per-Dandiset gear-wheel panel and stop disabling that gear in table view -- it was disabled because everything in it was plot-only, which is no longer true. Both entries now carry an info icon naming the view they apply to, and the panel is titled "Settings". Re-rendering a table also no longer discards the sort the user chose: toggling the new setting (or the binary/decimal prefix) rebuilt the table from scratch and snapped it back to the default column. The sort is now remembered per container element and column set, so it survives a re-render with new rows while a genuinely different table still starts from its own default. setSettingsBtnDisabled had no callers left and is removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts
Add 000068, 000144, 000299, 000411 and 000470, and split the list onto one ID per line now that it no longer fits comfortably on one. The unit tests now derive their fixture from the list and check its shape (six-digit IDs, no duplicates, in order) instead of restating it, and the changelog points at the constant rather than enumerating it, so adding an ID is a one-line change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds four new "scaled" usage metrics to the per-Dandiset table that normalize raw usage statistics by the actual content each Dandiset holds. These metrics provide better insight into usage patterns relative to dataset size.
Key Changes
scaled_metric,format_ratio): Calculate and format per-unit metrics with appropriate precision based on magnitudedecode_maybe_gzipped_responseandfetch_maybe_gzipped_textto handle client-side decompression of gzipped derivative files served withoutContent-Encodingheadersparse_dandiset_numbers_jsonlto parse Dandiset ID → number mappings (complementing the existing title parser)render_sortable_tableto handle NaN values (representing unavailable metrics) by always sorting them to the bottom, regardless of sort directionImplementation Details
DecompressionStreamAPI for efficient streaming decompression of large fileshttps://claude.ai/code/session_013Ym6LaKiE7zjkPMfzKMzts